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Abstract.  Process  performance  forms  the  cornerstone  of  the  high-maturity 
concepts  in  the  CM  Ml®.  High  maturity  generates  great  discussion  in  the  CMMI- 
based  process  improvement  world.  However,  understanding  process  performance 
provides  benefits  to  an  organization  whether  or  not  it  adopts  the  CM  Ml.  The  CM  Ml 
provides  a  framework  for  an  organization’s  process  improvement  efforts.  At  its 
highest  levels,  the  CM  Ml  describes  how  an  organization  can  use  process  perfor¬ 
mance  measures  to  understand  and  improve  its  business  processes.  While  this 
article  will  mention  the  high-maturity  process  areas  from  the  CM  Ml,  it  will  primarily 
focus  on  the  analysis  of  process  performance  to  help  an  organization.  Comments 
from  wise  men  and  women  throughout  history  illustrate  ideas  related  to  process 
performance.  In  one  of  his  songs,  Jimmy  Buffett  said,  “Chasing  illusions  can  get 
quite  confusing.1” 

This  is  true  of  the  use  of  process  performance  measures. 

To  understand  how  the  use  of  process  performance  measures 
affect  an  organization,  it  is  good  to  look  back  at  some  words  of 
wisdom  related  to  the  concepts  behind  the  use  of  performance 
measures  and  results.  This  article  will  illustrate  the  practical 
meaning  and  benefits  of  understanding  process  performance  by 
drawing  connections  to  famous  quotes. 

Setting  the  Foundation 

Managers  often  struggle  for  a  clear  understanding  of  what  is 
happening  on  their  projects.  They  find  themselves  in  the  same  situ¬ 
ation  as  Alexandre  Ledru-Rollin  when  he  said,  “There  go  my  people. 

I  must  find  out  where  they  are  going  so  that  I  can  lead  them.2” 

Good  use  of  process  performance  data  depends  on  estab¬ 
lishing  a  foundation  of  measurement  collection.  In  the  CMMI, 
this  starts  with  the  Measurement  and  Analysis  (MA)  process 
area.  In  MA,  an  organization  identifies  its  information  needs 
and  measurement  objectives.  Managers  are  always  looking 
for  information  to  help  them  answer  questions  like  Mr.  Ledru- 
Rollin.  By  starting  with  information  needs,  an  organization  can 
specify  what  is  needed  to  answer  some  of  those  management 
questions.  It  is  important  to  define  those  measures  clearly  so 
everyone  is  collecting  the  same  data,  the  same  way.  Operational 
definitions  of  measures  are  critical  to  measurement  success.  For 
example,  as  a  measure,  a  work  hour  can  represent  many  things. 
An  organization  should  define  what  it  needs  to  collect.  Is  it  a  di¬ 
rect  hour,  an  indirect  hour,  a  billable  hour  or  a  support  hour?  By 
clearly  defining  each  measure  an  organization  sets  itself  up  for 
more  accurate  and  meaningful  reporting.  Care  should  be  taken 


to  ensure  that  the  data  is  collected  accurately  and  analyzed 
appropriately.  Managers  should  also  communicate  the  results 
of  the  measurement  activities  back  to  the  people  collecting  the 
measures.  By  doing  so,  the  managers  give  the  practitioners  a 
stake  in  the  measurements.  The  measures  will  mean  more  to 
the  practitioners,  which  will  lead  to  more  accurate  reporting. 
Without  this  communication,  an  organization  ends  up  boxed  in 
as  Rowan  D.  Williams  stated,  “Bad  human  communication  leaves 
us  less  room  to  grow.3” 

Measurement  establishes  the  foundation  that  grows  into  the 
ability  to  use  process  performance  data  to  help  an  organization 
improve.  Inaccurate  reporting  stifles  that  growth. 

Dwight  D.  Eisenhower  said,  “Things  are  more  like  they  are 
now  than  they  have  ever  been  before.4” 

While  that  may  seem  obvious,  in  the  world  of  process  per¬ 
formance,  it  cannot  be  taken  for  granted.  In  order  to  know  how 
things  are  now,  an  organization  must  measure  the  current  state  of 
its  process  performance  and  compare  it  to  historical  performance. 
Each  of  the  high-maturity  process  areas  in  the  CMMI  contains 
practices  that  look  at  historical  results,  measure  current  perfor¬ 
mance,  forecast  future  performance,  and  look  to  make  improve¬ 
ments.  As  Philip  Crosby  said,  “Making  a  wrong  decision  is  under¬ 
standable.  Refusing  to  continually  search  for  learning  is  not.5” 

Organizational  Process  Performance 

As  mentioned  above,  an  organization  establishes  measure¬ 
ment  goals  based  on  information  needs.  As  the  organization 
accumulates  historical  measurement  data,  it  can  begin  to  predict 
process  performance  based  on  past  results.  In  the  Organization¬ 
al  Process  Performance  (OPP)  process  area  in  the  CMMI,  the 
organization  refines  those  goals  based  on  a  statistical  analysis 
of  historical  data  and  business  needs  for  quality  and  process 
performance.  This  is  important  because  as  Douglass  Lurtan 
pointed  out,  “When  you  determine  what  you  want,  you  have 
made  the  most  important  decision  of  your  life.  You  have  to  know 
what  you  want  in  order  to  attain  it.6” 

A  statistical  analysis  of  historical  data  is  necessary  to  validate 
these  goals  as  attainable.  Organizations  should  avoid  setting 
goals  like,  “We  want  to  be  a  world-class  provider  of  choice.”  No 
one  knows  what  that  means,  but  it  sounds  cool.  People  relate 
to  goals  like,  “We  want  to  reduce  customer  found  defects  by 
25%  in  the  next  year.”  Organizations  should  set  goals  that  are 
clear,  measureable,  realistic,  and  easy  to  understand.  After  the 
organization  sets  its  goals,  it  needs  to  identify  which  processes 
contribute  to  achieving  those  goals.  Organizations  should  not 
reach  for  too  much  in  analyzing  processes.  It  takes  time  and 
money  to  perform  quantitative  analysis.  Concentrate  on  those 
processes  that  are  of  concern  or  that  provide  the  most  insight 
into  the  achievement  of  business  needs.  There  must  be  busi¬ 
ness  reasons  for  choosing  the  processes  for  analysis. 

To  determine  if  they  can  attain  what  they  want,  an  organiza¬ 
tion  establishes  process  performance  baselines  to  understand 
past  performance  and  process  performance  models  to  predict 
future  behavior.  Keep  in  mind  that  these  are  just  tools  because 
as  H.  Thiel  said,  “Models  are  to  be  used,  not  believed.7” 

These  tools  give  insight  into  process  performance.  A  process 
performance  baseline  shows  an  organization  its  expected  range 
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of  performance  based  on  past  performance.  By  knowing  the 
expected  range  of  performance,  an  organization  understands 
whether  or  not  a  given  process  can  meet  its  performance  goals. 
Bertrand  Russell  said,  “The  degree  of  one’s  emotion  varies 
inversely  with  one’s  knowledge  of  the  facts— the  less  you  know, 
the  hotter  you  get.8” 

Without  the  facts,  managers  can  make  reactive,  emotional 
decisions.  Such  decisions  often  lead  an  organization  down  the 
wrong  path.  Understanding  expected  performance  reduces 
emotional  decisions  by  giving  managers  an  objective  view  and 
reasonable  performance  expectations.  Emotional  reaction  goes 
away  and  objective  decision  making  becomes  possible. 

Process  performance  models  allow  an  organization  to  explore  the 
relationships  between  different  pieces  of  their  process.  By  using  past 
performance  to  understand  how  the  different  parts  of  the  process 
relate  to  one  another,  organizations  can  begin  to  predict  what  will 
happen  in  later  parts  of  the  process  based  on  what  happens  in  an 
earlier  part  of  the  process.  This  gives  an  organization  understanding 
of  what  it  can  do,  not  just  what  it  has  done.  John  Wooden  stressed, 
“Do  not  measure  yourself  by  what  you  have  accomplished,  but  by 
what  you  should  have  accomplished  with  your  ability.9” 

Process  performance  models  enable  managers  to  understand 
their  ability,  and  understand  when  actual  results  vary  from  that 
ability.  In  the  CM  Ml,  process  performance  models  start  with  a 
controllable  factor,  like  project  size,  and  create  predictive  models 
based  on  the  understanding  of  the  effects  of  changes  to  that 
factor.  For  example,  an  organization  knows  that  a  size  increase 
of  more  than  1 0%  during  the  design  phase  causes  increases 
in  test  defect  rates.  Such  knowledge  can  be  used  to  determine 
if  additional  peer  reviews  or  testers  are  needed  to  accommo¬ 
date  the  size  change  and  prevent  a  significant  increase  in  test 
defects.  Other  models,  while  they  may  not  be  considered  process 
performance  models  in  CM  Ml  terms  can  also  help  organizations 
understand  and  manage  their  projects.  For  example,  if  an  organi¬ 
zation  knows  that  finding  a  higher  than  predicted  rate  of  require¬ 
ments  review  defects  historically  means  a  reduction  in  test  and 
customer-found  defects,  it  can  anticipate  performance  results  and 
make  decisions  related  to  those  future  lifecycle  phases. 

Quantitative  Project  Management  (QPM) 

In  QPM,  the  project  managers  within  the  organization  select 
the  measures  and  techniques  they  will  use  to  manage  process 
performance.  Sharon  Salzberg  stated,  “Each  decision  we  make, 
each  action  we  take,  is  born  out  of  intention.10” 

In  QPM,  the  measures  and  techniques  used  in  the  project  are 
selected  based  on  the  objectives  established  in  OPP  and  any 
unique  aspects  of  the  project.  If  there  is  no  connection  to  the 
organization’s  objectives,  the  organization  goes  back  to  chas¬ 
ing  illusions  or,  as  Bob  Seger  offered,  “Working  on  mysteries 
without  clues.11” 

Organizations  should  consider  which  items  to  include  and 
which  to  leave  out.  Joshua  Schachter  said  it  well  when  he  made 
the  point,  “Every  decision  has  a  cost.  Do  I  make  this  decision  at 
all  or  can  I  move  on  to  the  next  thing?  What  we  decided  to  leave 
out  is  almost  as  important  as  what  we  put  in.12” 

The  baselines  and  models  that  an  organization  creates  give 
insight  into  the  quality  and  performance  objectives  set  by  the  or¬ 


ganization.  As  Confucius  said,  “The  expectations  of  life  depend 
on  diligence;  the  mechanic  that  would  perfect  his  work  must 
first  sharpen  his  tools.13” 

Process  performance  baselines  and  models  provide  the  tools, 
which  an  organization  sharpens  over  time  as  it  gains  an  under¬ 
standing  of  its  process  performance.  But,  tools  must  be  used.  As 
Debra  Wilson  explains,  “People  who  do  not  use  the  tools  given 
to  them  only  injure  themselves.14” 

If  an  organization  does  not  make  use  of  QPM  tools,  it  loses  an 
opportunity  to  meet  business  goals  and  improve  performance. 
QPM  is  where  projects  use  the  models  and  baselines  that  are 
established  in  OPP  to  help  manage  their  projects. 

Process  Performance  and  Decisions 

Not  every  process  is  ripe  for  process  performance  measure¬ 
ment.  An  organization  should  concentrate  on  those  that  directly 
address  business  and  performance  objectives.  Start  with  a  small 
set  and  build  from  there.  Once  an  organization  understands 
its  past  results,  other  areas  of  opportunity  present  themselves. 
Organizations  must  start  somewhere,  because  as  Washington 
Irving  said,  “One  of  the  greatest  and  simplest  tools  for  learning 
more  and  growing  is  doing  more.15” 

When  projects  use  the  tools  available  to  them,  they  gain  in¬ 
sight  and  make  better  management  decisions.  When  the  actual 
performance,  or  prediction  of  performance,  does  not  match 
expectations  set  by  the  baselines  and  models,  managers  should 
ask  questions  and  take  action. 

Lee  lococca  said,  “If  I  had  to  sum  up  in  one  word  what  makes 
a  good  manager,  I  would  say  decisiveness.  You  can  use  the  fan¬ 
ciest  computers  to  gather  the  numbers,  but  in  the  end  you  have 
to  set  a  timetable  and  act.16” 

lococca  correctly  contends  that  numbers  are  not  answers. 
Numbers  represent  indicators  that  managers  should  use  to  ask 
questions  that  lead  to  better  decisions.  By  establishing  base¬ 
lines  and  models,  an  organization  sets  its  managers  up  to  make 
decisions  based  on  an  understanding  of  process  performance. 
Using  our  size  example  from  earlier,  if  an  organization  knows 
that  an  increase  in  project  size  of  more  than  1 0%  causes  a  cor¬ 
responding  increase  in  test  defects,  a  manager  can  adjust  staff 
levels  or  increase  test  time  to  allow  for  what  it  expects  based  on 
past  performance. 

Causal  Analysis  and  Resolution 

As  Crosby  pointed  out  in  the  earlier  quote,  organizations  must 
continually  learn  by  looking  at  their  past  mistakes  and  problems. 
That  concept  forms  the  basis  for  Causal  Analysis  and  Resolu¬ 
tion  (CAR).  Catherine  Aird  stated,  “If  you  cannot  be  a  good 
example,  then  you  will  just  have  to  be  a  horrible  warning.17” 

Both  good  examples  and  horrible  warnings  should  be  looked 
at  when  selecting  outcomes  for  analysis  in  CAR.  When  ana¬ 
lyzing  process  performance,  the  organization  should  look  at 
what  has  worked  well  in  addition  to  what  needs  improvement. 
Successes  should  be  leveraged  across  the  organization  and 
the  root  causes  of  problems  should  be  resolved  to  prevent  the 
recurrence  of  the  problem.  As  Chuck  Berry  told  us,  “Do  not  let 
the  same  dog  bite  you  twice.18" 

Often  organizations  focus  on  symptoms  rather  than  root 
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causes.  Getting  the  right  people  in  the  room,  which  means 
those  involved  in  the  process,  helps  identify  root  causes  of  prob¬ 
lems  or  successes.  Organizations  leverage  their  successes  by 
analyzing  the  causes  behind  them  just  as  they  fix  problems  by 
analyzing  the  causes  behind  them.  It  may  be  true  as  Mark  Twain 
said,  “Few  things  are  harder  to  put  up  with  than  the  annoyance 
of  a  good  example.19" 

However,  successes  can  create  peer  pressure  for  others  in 
an  organization  to  improve.  By  using  CAR,  an  organization  can 
identify  which  annoying  good  examples  are  worth  promulgating. 
On  the  flip  side,  it  is  also  true  that,  “The  best  way  to  escape  from 
a  problem  is  to  solve  it,20”  as  Alan  Saporta  pointed  out.  CAR 
allows  an  organization  to  find  root  causes  and  prevent  problems 
from  recurring  again  and  again  and  again  and  ... 

Organizational  Performance  Management 

Organizational  Performance  Management  (OPM)  asks  an 
organization  to  select  the  improvements  it  wants  to  make  and  to 
put  structure  in  place  to  deploy  and  analyze  improvement  pro¬ 
posals.  The  potential  improvements  can  come  from  a  variety  of 
sources.  One  source  is  when  a  project’s  results  historically  show 
that  they  cannot  reach  performance  goals.  For  example,  if  the 
goal  is  to  be  within  1 0%  of  estimates  and  the  project  is  always 
25%  to  40%  off,  the  project  is  unlikely  to  ever  meet  the  goal 
without  making  a  process  change.  Winston  Churchill  pointed 
to  this  when  he  said,  “Success  consists  of  going  from  failure  to 
failure  without  loss  of  enthusiasm.21” 

However,  that  success  only  comes  from  making  change.  The 
results  of  a  CAR  discussion  can  also  be  the  source  for  potential 
improvements.  For  CAR  groups  to  be  successful,  an  organiza¬ 
tion  must  provide  feedback  that  shows  the  results  are  consid¬ 
ered  important  and  that  the  results  are  being  used.  As  Colin 
Powell  said,  “The  day  soldiers  stop  bringing  you  their  problems 
is  the  day  you  have  stopped  leading  them.  They  have  either  lost 
confidence  that  you  can  help  them  or  concluded  that  you  do  not 
care.  Either  case  is  a  failure  of  leadership.22” 

Another  source  comes  from  looking  outside  the  organization 
for  innovations.  Organizations  often  become  enamored  with 
their  own  ideas  and  refuse  to  look  outside  of  themselves.  This  is 
the  trap  Friedrich  Nietzsche  spoke  of  when  he  said,  “Many  are 
stubborn  in  pursuit  of  the  path  they  have  chosen,  few  in  pursuit 
of  the  goal.23” 

All  available  information  and  sources,  internal  and  external 
to  the  organization  should  be  used  to  support  improvement 
initiatives.  Jimmy  Buffett  summed  this  up  when  he  told  us,  “I 
have  read  dozens  of  books  about  heroes  and  crooks,  and  I  have 
learned  much  from  both  of  their  styles.24” 

Business  goals  should  drive  organizational  improvements.  Out¬ 
side  ideas  can  be  just  as  valid  as  those  that  come  from  within. 

Validation  plays  an  important  role  in  OPM.  There  are  several 
ways  to  validate  if  an  improvement  is  successful.  These  include 
piloting  changes,  modeling  behavior  and  simulating  results.  To 
the  extent  possible,  improvements  driven  by  process  changes 
should  be  validated  statistically  to  ensure  that  observed  chang¬ 
es  are  not  random.  In  other  words,  a  quantitative  look  should 
be  taken  to  ensure  that  a  significant  change  has  occurred.  This 
prevents  the  pitfall  that  Mr.  Spock  addressed  when  he  said,  “A 
difference  that  makes  no  difference  is  no  difference.25” 


Whatever  method  is  chosen,  organizations  must  find  proj¬ 
ect  managers  willing  to  take  the  first  steps  in  trying  out  new 
improvements.  They  should  be  willing  to  follow  Frank  Zappa’s 
words,  “I  will  do  the  stupid  thing  first  and  then  you  shy  people 
follow.26” 

Improvement  proposals  do  not  always  work.  An  organization 
should  not  try  to  force  an  idea  because  it  seems  like  it  should 
work.  Use  the  validation  results  to  determine  if  the  change  is 
worth  adopting.  W.C.  Fields  explained  that  by  saying,  “If  at  first 
you  do  not  succeed,  try,  try  again.  Then  quit.  There  is  no  use  be¬ 
ing  a  damned  fool  about  it.27” 

However,  if  the  organization  determines  that  the  improvement 
was  a  success,  then  a  plan  should  be  put  in  place  to  deploy  it. 

Using  Process  Performance  Measures 
to  Manage  Change 

All  improvements  require  change.  Ideas  may  be  easily  under¬ 
stood  and  accepted,  but  change  always  comes  hard.  Charles 
Kettering  explained,  “The  world  hates  change,  yet  it  is  the  only 
thing  that  has  brought  progress.28” 

While  change  is  critical  for  improvement,  change  must  be 
managed.  Changes  deployed  to  the  organization  should  be  as 
timely  as  possible,  but  accomplished  in  an  orderly  fashion.  As 
John  Wooden  told  his  teams,  “Be  quick,  but  do  not  hurry.29" 

Unmanaged  change  creates  chaos,  but  managed  change  brings 
benefits.  As  Francis  Bacon  pointed  out,  “Things  alter  for  the  worse 
spontaneously,  if  they  are  not  altered  for  the  best  designedly.30” 

Organizations  have  to  deal  with  unplanned,  spontaneous 
change.  Managed  change  is  easier  to  accept  and  sets  the  foun¬ 
dation  for  future  improvement. 

Using  process  performance  measures  greatly  aids  an  organi¬ 
zation  in  making  improvements.  However,  organizations  should 
not  blindly  follow  the  numbers.  Numbers  can  be  manipulated  as 
Mark  Twain  said,  “Get  your  facts  first,  and  then  you  can  distort 
them  as  much  as  you  please.31” 

As  stated  previously,  numbers  are  just  indicators.  To  create 
useful  indicators,  organizations  should  clearly  define  the  analysis 
techniques  that  will  be  used  and  the  rationale  for  using  them. 
Organizations  must  never  lose  sight  of  the  fact  that  the  percep¬ 
tion  of  the  staff  is  just  as  important  as  the  numbers.  George 
Santayana  explained,  “Those  who  speak  most  of  progress 
measure  it  by  quantity  and  not  by  quality.32” 

Both  the  hard  numbers  and  soft  perceptions  determine  the 
success  of  any  improvement  effort.  If  the  numbers  look  good, 
but  the  people  have  legitimate  reasons  for  objection,  the  orga¬ 
nization  must  consider  their  viewpoint.  Malcolm  Gladwell  said  it 
well  when  he  explained  the  need  for  balance,  “Truly  successful 
decision  making  relies  on  a  balance  between  deliberate  and 
instinctive  thinking.33” 

Organizations  collect  a  lot  of  numbers,  but  real  value  comes 
when  they  are  used.  Establishing  a  measurement  foundation 
enables  the  use  of  process  performance  measures  once  an 
organization  builds  some  historical  data.  The  high-maturity  pro¬ 
cess  areas  in  the  CM  Ml  provide  guidance  on  how  quantitative 
information  and  process  performance  measures  can  be  used 
to  help  an  organization  meet  its  business  goals.  Remember  as 
Hesiod  stated  around  800  BC,  “Observe  due  measure,  for  right 
timing  is  in  all  things  the  most  important.34" 
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The  time  is  now  for  building  a  measurement  program  with  the 
vision  for  how  performance  measures  will  be  used.  Understand¬ 
ing  process  performance  can  be  perplexing.  Others  weathered 
the  storms  of  change  in  the  past.  In  order  to  plan  for  the  future 
of  process  improvements  and  make  meaningful  change,  organi¬ 
zations  should  consider  words  of  wisdom  from  those  who  came 
before  us.3> 


Disclaimer: 


CM  Ml®  is  registered  in  the  U.S.  Patent  and  Trademark  Office 
by  Carnegie  Mellon  University. 
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